Large Language Models (LLMs) have demonstrated remarkable capabilities in open-ended text generation tasks. However, the inherent open-ended nature of these tasks implies that there is always room for improvement in the quality of model responses. To address this challenge, various approaches have been proposed to enhance the performance of LLMs. There has been a growing focus on enabling LLMs to self-improve their response quality, thereby reducing the reliance on extensive human annotation efforts for collecting diverse and high-quality training data. Recently, prompting-based methods have been widely explored among self-improvement methods owing to their effectiveness, efficiency, and convenience. However, those methods usually require explicitly and thoroughly written rubrics as inputs to LLMs. It is expensive and challenging to manually derive and provide all necessary rubrics with a real-world complex goal for improvement (e.g., being more helpful and less harmful). To this end, we propose an ImPlicit Self-ImprovemenT (PIT) framework that implicitly learns the improvement goal from human preference data. PIT only requires preference data that are used to train reward models without extra human efforts. Specifically, we reformulate the training objective of reinforcement learning from human feedback (RLHF) -- instead of maximizing response quality for a given input, we maximize the quality gap of the response conditioned on a reference response. In this way, PIT is implicitly trained with the improvement goal of better aligning with human preferences. Experiments on two real-world datasets and one synthetic dataset show that our method significantly outperforms prompting-based methods.

A phrase like "It's cold today" would suggest a very different temperature if it were uttered in Nairobi or Montreal, while words like "troll" and "tweet" referred to totally different things just a generation ago. Although contemporary large-scale pretrained language models are very effective at learning linguistic representations, they are not as well equipped at capturing speaker/author-related temporal, geographical, social and other contextual aspects. In the new paper LMSOC: An Approach for Socially Sensitive Pretraining, a Twitter Cortex research team proposes LMSOC, a simple but effective approach for learning both linguistically contextualized and socially sensitive representations in large-scale language models. An implicit assumption in most pretrained language models (PLMs) is that language is independent of extra-linguistic contexts such as speaker/author identity and social settings. Despite the impressive achievements of PLMs, this remains a critical weakness, as there is strong evidence that socio-linguistics can significantly impact social context processing performance.